CMPT 825 : Natural Language Processing Spring 2008

Hidden Markov Models need a large set of parameters which are induced from a text-corpus. The parameters should be optimal in the sense that resulting models assign high probabilities to seen training data. There are several methods to estimate model parameters. The first one is to use each word as a state and estimate the probabilities using the relative frequencies. The second method is a variation of the first method. In this model, words are automatically grouped by similarity of distribution in the corpus. Each group is represented by a state in the model. The second method has the advantage of drastically reducing the number of model parameters and thereby reducing the sparse data problem. The third method uses manually defined categories. An important difference to the second method with automatically derived categories is that with manual definition a word can belong to more than one category. The fourth method is a variation of the third method and is also used for part of speech tagging. This method does not need a pre-annotated corpus for parameter estimation. The parameters are estimated using the Baum-Welch algorithm. This paper proposes a fifth method for estimating natural language models combining the advantages of the methods mentioned above.